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Executive  Summary 


This  paper  documents  two  small  manufacturing  enterprises’  (SMEs’)  efforts  to  select 
advanced  software  technologies  for  their  business  operations.  While  the  two  companies’ 
market  spaces  are  completely  different,  each  faced  business  and  operational  issues  that  are 
common  to  the  broad  SME  community.  Conducting  both  companies’  technology  selection 
efforts  concurrently  allowed  the  Technology  Insertion,  Demonstration,  and  Execution 
Program  to  address  a  wide  range  of  issues  and  better  leverage  the  selection  expense. 

The  generic  selection  methodology  used  was  a  downsizing  of  the  PEC  A  methodology 
augmented  by  Analytic  Hierarchy  Process  (AHP)  decision  support  (see  Appendix  A).  PECA 
was  developed  by  the  National  Research  Council  of  Canada  and  the  Carnegie  Mellon® 
Software  Engineering  Institute.  The  body  of  this  report  describes  the  companies,  the  process, 
the  issues,  and  the  lessons  learned  during  the  software  selection.  The  lessons  taught  us  how 
important  it  is  for  SMEs  to 

•  understand  their  business  and  how  the  proposed  software  will  support  their  firm’s 
growth  strategy 

•  develop  or  use  a  process  to  assign  tasks  and  involve  stakeholders 

•  if  necessary,  involve  specialists  in  decision  support  and  technology  adoption  to  help 
clarify  issues  and  identify  potential  pitfalls 

•  investigate  vendors  and  their  software  offerings  from  a  variety  of  perspectives 


Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  office. 
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Abstract 


Small  manufacturing  enterprises  (SMEs)  face  a  number  of  challenges  when  selecting  and 
implementing  advanced  software  technologies.  These  challenges  may  include  the  lack  of 
awareness  of  the  specific  technologies  and  commercial  products  available,  the  lack  of  ability 
to  select  the  appropriate  product,  and  the  lack  of  skill  sets  needed  to  utilize  the  selection 
techniques. 

This  paper  documents  the  actual  process  and  benefits  of  advanced  software  technologies 
adoption  by  two  SMEs.  Considerations  for  defining  requirements  and  selecting  a  software 
product  are  described.  This  technical  note  explains  the  issues  involved  for  SMEs,  presents 
methods  they  can  use,  and  provides  artifacts  used  in  this  documented  case. 
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1  Introduction 


1.1  Background 

Small  manufacturing  enterprises  (SMEs)  today  are  faced  with  many  challenges:  among  them  are 
global  competition,  volatile  markets,  and  rapidly  evolving  technology.  These  challenges  require 
SMEs  to  raise  their  performance  to  new  levels.  SMEs  must  operate  with  increased  efficiency  to  meet 
the  demands  of  global  competition.  They  must  reengineer  their  processes  to  reduce  the  time  to  market 
for  new  products.  They  must  also  continually  improve  their  products  and  services  to  meet  ever- 
increasing  customer  demands.  Advanced  Information  Technology  (IT)  tools  such  as  computer-aided 
engineering  (CAE)  and  integrated  manufacturing  execution  systems  (MESs)  can  help  SMEs  achieve 
these  goals. 

The  indications  are,  however,  that  many  SMEs  have  yet  to  adopt  advanced,  commercial  off-the-shelf 
(COTS)  software  tools.  For  example,  the  Air  Force  white  paper,  Initiative  for  Small  and  Medium 
Enterprises ,  quoted  a  study  of  1,002  companies.  According  to  the  study,  35%  of  companies  with  50 
or  fewer  employees  had  a  computer-aided  design  and  engineering  capability.  For  companies  with 
500  employees  or  more,  the  figure  was  85%  [Boden  99].  Similarly,  a  survey  of  200  SMEs  in 
southwestern  Pennsylvania  found  that 

•  28%  have  solid  modeling  capabilities. 

•  23%  have  simulation  capabilities. 

•  16%  use  Finite  Element  Analysis. 

•  Fewer  than  30%  communicate  directly  to  suppliers  and  customers  over  the  Internet  [Catalyst  02]. 

SME  reluctance  to  adopt  advanced  software  technologies  may  be  attributed  to  various  factors, 
barriers,  and  constraints.  These  factors  include  the  perception  that  advanced  software  represents  a 
cost,  not  an  asset;  the  lack  of  knowledge  of  the  technologies;  the  lack  of  financial  resources;  and  the 
lack  of  expertise  in  technology  adoption.  The  Technology  Insertion,  Demonstration,  and  Evaluation 
(TIDE)  Program  investigated  these  issues  by  collaborating  with  two  SMEs  to  specify  and  select 
advanced  software  technologies.  In  this  effort  the  TIDE  program  and  the  Software  Engineering 
Institute  also  collaborated  with  the  Duquesne  University  Institute  for  Economic  Transformation  (IET) 
and  the  National  Institute  of  Standards  and  Technology  (NIST). 

1.2  Magdic  Precision  Tooling,  Incorporated 

Magdic  Precision  Tooling,  Incorporated  designs  and  manufactures  sophisticated  compaction  tooling 
for  the  powder  metal  industry.  Over  several  years,  Magdic  has  worked  with  the  Duquesne  University 
IET  to  implement  strategic  business  planning,  cross-training,  process  flow  optimization,  and  other 
continuous  improvement  activities.  To  maintain  its  growth  and  profitability,  the  company  began 
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looking  at  technology  improvement.  Specifically,  the  firm  identified  the  following  areas  of  concern 
regarding  its  design  and  manufacturing  process: 

•  The  total  cycle  time  was  greater  than  desired. 

•  An  opportunity  existed  for  manufacturing  process  improvement. 

•  Processes  were  paper  based. 

•  Retrieval  of  legacy  information  was  difficult. 

•  Revision  control  of  drawing  sets  was  manual. 

•  Shop  scheduling  was  not  optimized. 

•  Shop  capacity  was  difficult  to  monitor. 

These  issues  are  typical  of  the  “sneaker  net”  communication  scenario  found  in  many  SMEs. 
Processes  and  tasks  are  described  verbally.  Job  orders  are  delivered  by  hand  throughout  the  shop 
floor.  If  paper  documentation  exists,  it  is  often  incomplete  or  out  of  date. 

Magdic  personnel  felt  that  electronic  data  display  tools  could  enhance  data  management,  improve 
documentation,  lead  to  parallel  job  processing,  and  ultimately  help  to  reduce  product  cycle  time.  At 
the  same  time,  these  improvements  would  help  Magdic  to  enhance  its  position  as  a  market  leader  for 
rapid  product  design  and  delivery.  IET  consultants  subsequently  matched  Magdic ’s  improvement 
need  with  the  technology  adoption  research  being  performed  by  the  TIDE  Program. 

1.3  Gentile  Manufacturing  Company 

Gentile  Manufacturing  Company,  Incorporated  designs  and  manufactures  sophisticated  parts  and 
assemblies.  Gentile  had  all  of  Magdic’s  challenges  plus  the  following  unique  issues: 

•  Cost  tracking  was  manual  and  not  progressive. 

•  The  quotation  process  was  lengthy  and  difficult. 

•  A  key  client  required  real-time  visibility  of  work  status. 

•  The  raw  material  inventory  was  not  tracked. 

Most  importantly,  Gentile’s  largest  customers  were  demanding  that  the  company  conduct  business  in 
a  “pure”  electronic  format.  The  ability  to  electronically  manage  and  integrate  business  and 
operational  data  would  enable  Gentile  to  respond  to  this  demand. 

1.4  Project  Motivation 

From  the  perspective  of  the  TIDE  Program,  the  ability  to  help  two  different  companies  adopt  a 
common  technology  solution  had  a  number  of  advantages.  It  would  allow  the  TIDE  Program  to 
increase  the  amount  of  technology  adoption  data  gathered  while  leveraging  program  resources.  It 
would  enable  TIDE  personnel  to  acquire  information  about  two  different  types  of  manufacturing 
businesses.  (Magdic  specializes  in  low-volume,  custom  products.  Gentile  was  a  higher  volume  job 
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shop.)  Finally,  it  would  enable  the  TIDE  Program  to  measure  the  impact  of  the  software  on  both 
businesses  going  forward. 


1 .5  Project  Scope 

As  initially  conceived,  the  effort  to  document  the  process  and  benefits  of  adopting  a  manufacturing 
execution  system  (MES)  would  be  conducted  in  three  phases:  Discovery  and  Planning;  System 
Implementation;  and  System  Analysis  and  Publication.  However,  shortly  after  completing  the 
Discovery  and  Planning  Phase,  Gentile  lost  a  large  customer.  That  loss  along  with  continued 
weakness  in  the  metalworking  market  forced  the  company  to  reorganize  its  business.  As  a  result, 
Gentile  opted  to  not  continue  with  the  implementation  phase  of  the  project.  Magdic  Precision  Tooling 
implemented  the  software  technology  that  was  selected.  Magdic’s  implementation  effort  will  be 
documented  in  a  future  technical  note. 

The  remainder  of  this  paper  describes  the  specification  and  selection  effort,  and  presents  lessons 
learned  to  help  other  SMEs  streamline  their  technology  selection  efforts. 


CMU/SEI-2003-TN-020 


3 


2  Case  Study 


During  the  Discovery  and  Planning  Phase,  TIDE,  Magdic,  and  Gentile  personnel 

1 .  performed  a  needs  assessment  and  business  case  analysis 

2.  established  a  selection  team  and  selection  process 

a.  determined  prospective  COTS  solutions  and  compared  them  to  system  requirements 

b.  demonstrated  products  and  selected  the  appropriate  software  and  hardware 

c.  procured  products 

2.1  Needs  Assessment  and  Business  Case  Development 

To  participate  in  the  TIDE  Program,  Magdic  submitted  a  technology  adoption  proposal.  TIDE 
personnel  reviewed  the  proposal,  compared  the  proposal  to  Magdic ’s  growth  strategies,  and  evaluated 
Magdic’s  ability  to  implement  the  proposal.  Gentile  was  introduced  to  the  TIDE  Program  through  the 
TIDE  workshop  “Introduction  to  Ecommerce  for  SMEs”  conducted  in  December  of  2001 . 

In  the  case  of  Magdic  Precision  Tooling,  consultants  from  the  Duquesne  University  IET  had 
previously  helped  Magdic  to  implement  a  series  of  business  process  improvement  activities.  These 
changes  resulted  in  a  20%  increase  in  capacity  without  an  increase  in  overhead.  Magdic’s  strategy 
was  to  continue  improving  workflow  to  further  reduce  delivery  times,  enhance  customer  service,  and 
obtain  a  competitive  advantage.  The  company  wanted  help  implementing  a  computer-based  system 
for  controlling  job  information.  That  system  would  allow  Magdic  personnel  to 

•  scan  and  store  drawings  electronically 

•  enter  and  save  job  information  with  electronic  order  files 

•  display  drawings  along  with  the  latest  changes  at  each  machine 

•  ret  lie  vc  archived  job  files 

•  integrate  engineering  data  with  electronic  order  information 

The  consultant  from  the  Duquesne  University  IET  helped  Magdic  develop  a  business  case.  Based  on 
an  investment  of  $70K,  company  officials  predicted  a  10%  increase  in  capacity  and  a  30%  reduction 
in  cycle  time  on  new  tool  sets,  resulting  in  a  capacity  to  add  $200K  in  new  sales  annually.  Magdic 
presented  this  business  case  in  its  proposal. 

After  reviewing  Magdic’s  proposal,  TIDE  personnel  recommended  that  Magdic  expand  it  to  cover  an 
integrated  MES  that  would  provide  the  desired  capabilities  while  linking  accounting,  billing,  and 
other  front-office  functions.  It  would  also  enable  customers  to  review  the  status  of  their  orders  in  real 
time.  When  fully  implemented,  the  MES  could  help  Magdic  to  establish  a  virtually  paperless 
manufacturing  environment. 
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In  the  case  of  Gentile  manufacturing,  three  customers  had  asked  the  company  to  set  up  and  maintain 
electronic  Web  portals.  TIDE  staff  members  invited  experts  from  NIST  to  explore  the  possibility  of  a 
one-to-many  portal  translator  or  some  level  of  automation  to  assist  in  the  maintenance  of  these 
portals.  While  some  one-to-many  portal  automation  was  possible,  NIST  specialists  concluded  that 
Gentile  lacked  the  basic  manufacturing  execution  software  needed  to  automate  a  portal  translation. 

An  MES  could  provide  that  capability.  In  addition,  Gentile  had  an  opportunity  to  take  over  the 
renewal  parts  manufacturing  business  for  another  company.  This  activity  also  would  require  an  MES 
to  manage  the  volume  of  business.  While  no  formal  business  plan  was  developed,  the  new  business¬ 
forcing  function  justified  Gentile’s  interest. 

2.2  Establish  Selection  Team  and  Selection  Process 

In  the  next  step,  TIDE  staff  members  worked  with  Magdic  and  Gentile  personnel  to  identify  roles  and 
responsibilities  for  participants  and  to  develop  a  process  for  software  selection.  Based  on  their 
experience  in  analyzing  and  specifying  COTS  systems,  TIDE  specialists  suggested  using  the  PECA 
methodology.2  PECA  was  jointly  developed  by  the  National  Research  Council  Canada  and  the 
Carnegie  Mellon  "  Software  Engineering  Institute  (SEI)  to  evaluate  COTS  software,  document  the 
factors  involved,  and  record  the  decision-making  process. 

Initially,  the  TIDE  specialists  proposed  that  Magdic  and  Gentile  employees  form  a  joint  selection 
team.  TIDE  staff  members  would  train  team  members  on  the  PECA  process  and  document  their 
efforts.  However,  the  lack  of  time  and  available  SME  personnel  forced  the  TIDE  team  to  change  the 
strategy.  Instead  of  training  a  software  evaluation  team,  TIDE  members  ended  up  serving  on  it. 
Selection  team  members  included  Charles  Buhman,  Bill  Anderson,  and  Grace  Lewis  from  the  SEI;3 
Todd  Sterlitz,  Vice-President,  Gentile  Manufacturing  Co.;  Joe  Magdic,  President,  Magdic  Precision 
Tooling;  and  Simon  Frechette,  NIST. 

In  its  first  activity,  the  team  tried  to  define  the  goals  and  scope  of  evaluation.  The  team  struggled  with 
“scope”  verses  “goal”  semantics,  wasting  time  in  the  process.  Eventually,  the  team  agreed  on  the 
following: 

Scope:  Evaluate  a  small  set  of  software  packages,  hardware,  and  infrastructure  to  support  shop 
floor  control,  visualization,  and  a  paperless,  Internet-enabled  environment. 


2  PECA  is  a  COTS  selection  methodology.  The  name  is  taken  from  the  first  letter  of  each  of  the  process 
steps:  Plan,  Establish  criteria.  Collect  data,  and  Analyze  data. 

3 

This  touches  on  a  fundamental  difficulty  with  this  type  of  research:  participation  verses  observation.  It  is 
difficult  to  draw  conclusions  about  how  large  a  good  team  must  be.  In  this  case,  TIDE  staff  members  felt 
compelled  to  join  to  provide  needed  mass.  For  their  part,  the  SMEs  preferred  buying  subject  matter 
expertise,  rather  than  receiving  training  or  facilitation. 
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Goal:  Select  software  that  satisfies  the  needs  of  two  small  manufacturers  and  considers  the  shop  floor 
environment,  stakeholders’  needs,  ecommerce,  shop  floor  visualization,  and  collaboration  capability. 

These  statements  are  not  significantly  different.  The  teams  suggests  that  the  scope  and  goals  must  be 
precisely  defined  and  clearly  differentiated  to  avoid  wasting  time  on  semantics.4 

Next,  the  team  identified  a  series  of  tasks  and  a  schedule  for  completion.  This  too  required  discussion 
and  negotiation.  For  example,  SME  managers  Joe  Magdic  and  Todd  Sterlitz  simply  did  not  have  the 
time  to  take  advantage  of  external  resources  such  as  the  SEI  software  demonstration  laboratory.  They 
also  did  not  have  time  to  travel  across  town  to  the  SEI  facility.  And  they  needed  to  limit  the  time  that 
they  and  their  employees  could  commit  to  this  effort.  The  time  factor  remained  an  issue  throughout 
the  demonstration  project,  and  a  number  of  activities  were  modified  to  expedite  matters. 

2.2.1  Software  Requirements  Specification 

This  task  involved  identifying  the  fundamental  specifications  of  any  MES.  The  team  agreed  on  a 
number  of  fixed  requirements.  These  included 

•  budgets  for  purchase  and  implementation 

•  limits  on  the  training  time  and  effort  that  the  MES  would  require 

•  an  ability  to  implement  the  system  within  time  deadlines  imposed  by  customers. 

Additional  fixed  requirements  stated  that  the  selected  system  would  have  to 

•  be  PC  based 

•  use  Windows  NT/XP/2000 

•  be  compatible  with  Peachtree  software 

•  meet  key  customer  criteria  (e.g.,  electronic  collaboration  and  ecommerce  capabilities) 

•  support  AutoCAD  and  Unigraphics  packages 

Early  on,  the  team  dropped  the  requirement  for  compatibility  with  Peachtree  software  because  each 
candidate  MES  featured  an  internal  accounting  package. 

Having  identified  an  initial  set  of  selection  criteria,  the  team  discussed  the  need  to  interview 
stakeholders  from  accounting,  purchasing,  engineering,  quality,  production,  technology,  shipping,  and 
receiving.  The  goal  was  to  solicit  lower  level  requirements.  In  the  end,  however,  only  the  accounting 
stakeholder  was  interviewed.  Joe  Magdic  and  Todd  Sterlitz  provided  the  additional  input  to  save  the 


Goal  is  a  statement  of  the  desired  state.  Scope  is  an  agreement  on  boundary  conditions.  For  example,  if  the 
Goal  is  to  “make  money,”  how  ethically  it  is  earned  could  be  a  scope  issue.  The  authors  suggest  combining 
the  two  concepts  into  one  statement,  for  example,  “to  make  money  in  an  ethical  manner,”  to  focus  the 
discussion  on  the  project  and  not  semantics. 
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selection  team  valuable  time.  Joe  and  Todd  were  familiar  with  both  manufacturing  and  business 
operations,  and  had  previous  experience  implementing  computer-based  systems. 

In  general,  the  criteria  covered  four  areas: 

1 .  functionality 

2.  ability  to  integrate  with  legacy  systems 

3.  adoptability 

4.  strength  of  the  vendor 

Under  each  area,  the  team  listed  specific  requirements.  For  example,  “adoptability”  included  having 
Windows  conventions,  being  very  intuitive,  and  having  a  short  learning  curve. 

The  team  used  a  decision  support  tool  to  weigh  and  prioritize  the  requirements.  The  tool  helped 
facilitate  communication  among  team  members.  At  the  same  time,  it  provided  a  yardstick  to  measure 
candidate  MES  packages,  and  also  allowed  the  team  to  analyze  the  impact  of  decisions  on  candidate 
software  packages.  This  “sensitivity  analysis”  feature  became  important  later  in  the  evaluation. 

2.2.2  Comparing  Alternative  Software  Packages 

TIDE  members  from  NIST  investigated  MES  packages  (see  Appendix  D)  and  CAD  viewer  packages 
[Stevens  03].  In  addition,  the  selection  team  researched  the  Internet,  reviewed  trade  publications,  and 
informally  polled  SME  employees  and  customers.  Based  on  that  input,  the  team  developed  a  short 
list  of  four  MES  packages.  Figure  1  illustrates  the  process  they  used  to  compare  the  packages. 

The  selection  team  interviewed  candidate  vendors  for  first-pass  fit  against  the  requirements.  Each 
package  appeared  to  meet  the  requirements.  It  became  a  matter  of  judgment  as  to  how  well,  how 
easily,  how  quickly,  and  so  forth,  each  package  would  perform.  Furthermore,  each  vendor  featured 
local  support,  a  large  base  of  installed  systems,  and  an  active  user  community. 

Next,  the  team  asked  for  product  demonstrations.  All  the  product  vendors  could  provide  interactive 
live  demonstrations  of  their  systems  via  Internet  remote-session-viewing  technology  (WebEx™  in  this 
case).  The  vendors  ran  their  software  on  their  local  systems  while  the  evaluation  team  watched  live 
via  the  Internet.  This  eliminated  one  candidate  MES;  the  team  felt  that  the  package  did  not  fit  the 
underlying  make-to-order  business  process. 

The  selection  team  asked  the  remaining  three  candidate  vendors  to  bid  to  a  representative  system.  The 
three  bids  were  compiled  into  a  spreadsheet  that  grouped  costs  into  equivalent  categories.  (See  Table 
2,  page  56)  The  spreadsheet  included  one  aspect  of  life-cycle  cost  (annual  maintenance  fees)  to 
indicate  operational  cost. 


WebEx  is  a  trademark  of  WebEx  Communications.  Inc. 
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Figure  1:  Comparing  Alternate  Software  Packages 


8 


CMU/SEI-2003-TN-020 


On  the  surface,  the  price  between  the  high  and  low  bids  differed  by  less  than  seven  percent. 

However,  each  vendor  took  a  different  path  to  reach  the  net  price.  One  package  carried  a  high  list 
price  that  was  deeply  discounted  and  that  served  as  the  base  against  which  a  multiplier  was  applied  to 
derive  annual  maintenance  fees.  In  this  first  case,  a  high  list  price  resulted  in  high  annual 
maintenance  fees  ($5,400  per  annum).  The  next  package  had  a  low  list  price,  offered  no  discount,  but 
used  a  high  multiplier  to  derive  the  highest  annual  maintenance  fees  ($5,900  per  annum).  The  third 
package  had  a  list  price  in  the  middle  that,  when  extended  by  the  multiplier,  produced  the  lowest 
annual  maintenance  fees  ($2,600  per  annum).  When  these  recurring  fees  are  considered  over  a  few 
years  they  became  a  significant  cost  differentiator. 

Another  large  variable  was  the  cost  and  recommended  levels  of  training  and  start-up  assistance.  The 
recommended  packages  varied  from  eight  days  of  consulting  plus  two  distance-learning  classes  for 
$11,300,  six  days  of  consulting  at  $7,200,  and  three  days  of  consulting  plus  unlimited  factory  and 
Web-based  training  for  $2,900. 

The  team  cut  the  short  list  to  two  vendors  and  then  checked  those  vendors’  references.  Using  the 
Analytical  Hierarchy  Process  (AHP)  tool,  a  method  for  prioritizing  decisions  by  incorporating 
relevant  decision  criteria,  the  team  evaluated  and  compared  the  two  vendors’  software  packages,  and 
conducted  a  sensitivity  analysis.  (See  Appendix  A.)  That  analysis  raised  questions  about  whether 
either  vendor’s  package  could  provide  paperless,  shop  floor  control.  This  prompted  the  selection  team 
to  ask  for  a  second  round  of  demonstrations.  One  vendor  was  unable  to  demonstrate  the  requested 
functionality,  despite  earlier  claims  that  the  product’s  current  version  could  provide  it.  As  a  result,  that 
vendor — which  had  been  the  leading  candidate — lost  the  selection  team’s  confidence  and  the  sale. 

The  second  vendor  readily  admitted  that  this  functionality  was  new,  and  although  the  vendor  was 
confident  that  the  software  could  provide  paperless  shop  floor  control,  the  vendor  could  not  provide  a 
reference  that  could  vouch  for  that  functionality;  Magdic  would  be  the  first  company  to  use  it.  The 
selection  team  appreciated  that  vendor’s  frankness  and  awarded  it  the  contract. 

The  second  set  of  demonstrations  put  the  project  behind  schedule.  However,  it  validated  the  benefit  of 
the  decision-support  software  and  the  structured  COTS  selection  process  (and  verified  the  importance 
of  demos  to  validate  vendor  claims). 

2.2.3  Hardware  Considerations 

The  implementation  of  an  MES  at  Magdic  required  a  system  server,  a  large  format  scanner,  and  10 
terminals,  one  at  each  shop  floor  station.  TIDE  team  members  qualified  hardware  with  the  software 
vendor’s  and  Magdic’s  concurrence.  The  TIDE-supplied  hardware  was  purchased  following  SEI 
equipment  guidelines,  leveraging  the  university’s  purchasing  agreements  with  an  existing  supplier.  A 
mix  of  PCs  and  less  expensive  thin  client  workstations  for  the  shop  floor  was  purchased.  Magdic 
purchased  the  Uninterruptible  Power  Supply  (UPS),  firewall,  and  network  upgrades.  A  third-party 
network  installer  conducted  a  site  survey  and  added  several  network  drops  and  a  cabinet  to  house  the 
server  and  UPS. 
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2.2.4  Procurement  and  Licensing  Issues 

Procurement  was  much  more  complicated  than  originally  anticipated.  Despite  the  advice  of  vendors, 
certified  Microsoft  service  representatives,  and  several  experienced  IT  personnel,  TIDE  members 
were  unable  to  discern  the  correct  licensing  requirements  for  the  MES.  Some  products  (the  thin  client 
terminals)  required  Certified  Application  Licenses.  Others  (the  PC  thin  client  emulators)  came  pre¬ 
licensed  and  were  compatible  with  the  terminal  services  environment.  This  was  counterintuitive  as 
the  PCs  (which  have  a  stand-alone  utility)  came  with  an  embedded  license  for  the  relatively  less 
popular  terminal  services  mode  of  operation,  while  the  thin  client  terminals  that  only  have  utility  in 
that  mode  required  a  separate  license  expenditure.  This  situation  made  projecting  costs  difficult.  For 
example,  the  price  of  backup  software  tripled  when  its  license  incompatibility  was  finally  resolved. 

Furthermore,  the  maintenance  contract  provided  by  the  MES  vendor  did  not  cover  the  integrated 
third-party  packages.  When  one  of  these  packages  publicly  announced  feature  updates,  the  selection 
team  learned  that  it  did  not  have  the  right  to  them.  This  raises  a  number  of  concerns. 

•  If  a  company  purchases  additional  software  licenses,  which  versions  of  the  third-party  software 
are  included?  Are  these  versions  compatible? 

•  When  the  third  party  drops  support  of  a  given  version,  will  the  vendor  take  over? 

•  Who  is  responsible  for  purchasing  updates?  One  would  assume  that  the  vendor  would  provide 
them  under  the  maintenance  agreement,  but  this  was  not  the  case. 

The  vendors’  recommendation  to  purchase  separate  upgrades  for  the  embedded  third-party  products 
generated  a  set  of  issues  as  well. 

•  If  the  SME  buys  upgrades  separately,  what  level  of  coordination  through  the  base  vendor  is 
available? 

•  Does  the  SME  have  the  legal  right  (without  harming  its  ability  to  get  continued  support  from  the 
base  vendor)  to  integrate  a  revision  of  third-party  software  into  the  suite? 

•  Will  the  base  vendor  test  and  notify  compatibility  with  future  third-party  revisions? 

•  Does  the  vendor  supply  a  clear  interface  specification  and  instructions  for  installing  the  third- 
party  software?  Does  the  third  party  sanction  the  practice  and  procedure? 

These  are  not  just  theoretical  considerations.  When  the  MES  vendor  had  difficulty  correcting  a  fault 
in  a  Web  viewer  portal,  Magdic  was  willing  to  hire  a  third-party  expert  to  remedy  this  fault.  However 
the  license  restricted  reverse  engineering,  derivative  work,  and  remedial  software  repairs. 

The  solution  is  to  establish  a  sound  vendor  relationship  and  to  make  sure  company  needs  align  with 
the  vendor’s  target  market,  so  that  the  vendor  will  want  to  address  (or  at  least  not  ignore)  customer 
requests. 

2.2.5  Tools  and  Artifacts 

As  part  of  the  software  selection  process,  TIDE  staff  members  applied  a  number  of  artifacts: 
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1)  the  specific  hierarchical  requirements  tree  that  guided  our  product  evaluation  (see  Appendix  B). 

2)  a  product  dossier  document  that  originated  from  the  SEI  Evolutionary  Process  for  Integrating 
COTS -based  systems  (EPIC)  [Albert  02].  A  product  dossier  highlights  a  broad  range  of 
product  evaluation  issues  (see  Appendix  C). 

For  information  on  EPIC  see  <http://www.sei.cmu.edu/cbs>. 

3)  a  cost  comparison  spreadsheet  (see  Appendix  D). 

4)  a  Manufacturing  Execution  Systems  Product  Survey,  a  product  comparison  matrix  to  aid  in 
the  software  selection  process  (see  Appendix  E). 
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3  Lessons  Learned 


Budget  and  time  limitations,  a  bewildering  array  of  products,  and  lack  of  expertise  can  pose 

serious  challenges  to  SMEs  interested  in  adopting  advanced  software.  The  TIDE 

demonstration  responded  to  these  issues  by  emphasizing  both  preparation  and  process.  The 

TIDE  Program  offers  the  following  guidance  for  selecting  advanced  software  technologies: 

•  The  size  of  the  company  will  determine  the  type  and  amount  of  process  required.  With 
fewer  than  30  employees  each,  both  Magdic  and  Gentile  lacked  the  “mass”  needed  for 
the  formal  PECA  methodology,  forcing  the  process  to  adapt  to  the  circumstances. 
However,  the  participants  confirm  that  some  structure  was  necessary  to  move  the 
software  selection  process  forward. 

•  Team  composition  affects  team  tasking.  If  the  top  decision  makers  are  on  the  team,  tasks 
that  are  motivated  by  upward  communication  and  authority  enablement  are  less 
important,  if  not  unnecessary.  Top-level  management  in  a  small  enterprise  is  also  well 
founded  in  comprehensive  business  process  knowledge,  reducing  the  discovery  value  of 
non-team  stakeholder  involvement.  Stakeholder  involvement  becomes  more  motivated 
toward  buy-in,  training,  and  user  acceptance. 

•  Beware  the  PowerPoint®  demonstration.  When  a  vendor  switched  from  a  live  WebEx 
demonstration  to  a  canned  PowerPoint  slide  show,  vaporware5  was  soon  uncovered. 

•  Decision-support  software  can  be  very  helpful  for  software  selection  and  other  issues. 
Properly  implemented,  decision-support  software  can  help  rank,  compare,  and  clarify 
subjective  issues,  improve  communications  among  different  stakeholders,  and  facilitate 
the  “what  if’  thinking  that  can  lead  to  better  decisions.  However,  the  key  to  efficient  use 
of  this  software  is  in  limiting  the  scope  of  the  investigation.  Joe  Magdic  felt  that  an  SME 
user,  before  using  the  software  with  confidence,  would  need  someone  to  guide  the 
process  several  times. 

•  Stay  in  the  vendor 's  sweet  spot.  Finding  a  vendor  that  knows  and  is  committed  to  the 
SME’s  business  is  critical.  If  the  vendor  is  committed  to  the  SME’s  market,  the  SME’s 
issues  will  be  market  issues,  creating  more  incentive  for  the  vendor  to  resolve  them.  In 
addition,  there  may  be  other  users  who  have  already  addressed  common  questions  and 
issues. 

•  SMEs  must  do  their  homework.  Often,  vendors  and  prospective  customers  focus  on  the 
“bells  and  whistles”  of  the  software,  rather  than  the  “nuts  and  bolts.”  Once  that  software 


PowerPoint  is  a  registered  trademark  of  Microsoft  Corporation. 

5  Vaporware  is  defined  as  “products  announced  far  in  advance  of  any  release  (which  may  or  may  not 
actually  take  place)”  [Jargon  File  01]. 
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has  been  installed,  however,  “nuts  and  bolts”  features  become  very  important.  With  so 
much  on  the  line,  SMEs  need  to  learn  as  much  as  they  can  about  the  software’s 
capabilities,  compatibilities,  and  processes.  This  requires  the  SME  to  do  more  than  check 
references.  Ideally,  the  SME  should  visit  and  interview  customers  who  have  similar 
operations,  if  possible.  The  SME  should  require  the  vendor  to  demonstrate  typical  or 
critical  tasks. 

•  Trainers  should  have  domain  expertise.  Pay  attention  to  the  trainer’s  background  and 
domain  expertise  before  you  engage.  The  accounting  functions  generally  demand  a  deep 
background  and  understanding  of  accounting  and  how  the  software  operates.  This  is  not 
the  same  knowledge  that  it  takes  to  understand  the  operations  on  the  shop  floor. 

•  Vendors  will  sell  flexibility.  The  marketplace  forces  the  vendor  to  be  all  things  to  all 
people  (or  at  least  a  broad  enough  set  of  people  to  generate  a  market).  But  in  reality  the 
software  will  have  “optimal  use  scenarios” — those  ways  of  using  the  system  that  are  tried 
and  true.  These  are  the  scenarios  that  will  have  the  lowest  implementation  risk;  SMEs 
should  find  them  and  change  their  practices  to  take  advantage  of  them. 

•  The  SME  must  be  prepared  to  change.  COTS  software  is  designed  around  a  general 
business  model.  In  most  cases,  the  SME  will  have  to  modify  its  business  and  operational 
processes  to  use  the  software.  To  minimize  the  changes,  the  SME  should  select  a  package 
that  fits  its  needs  and  follows  the  way  it  does  business.  Still,  the  SME  should  expect  that 
changes  will  be  necessary  and  desirable,  especially  if  the  software  embodies  improved  or 
“industry  best”  practices.  This  also  will  keep  the  SME  closer  to  the  vendor’s  sweet  spot. 

•  Ask  questions  and  more  questions.  Such  questions  include  the  following: 

•  What  3rd  party  packages  are  bundled  in  the  software  suite  and  will  the  vendor 
support  them? 

•  What  is  involved  in  converting  legacy  data  to  work  with  the  new  system? 

•  Does  the  vendor  have  a  mechanism  to  educate  employees  about  the  optimal 
process  scenarios  to  leverage  his  software? 

•  Do  you  need  editable  versions  of  (and  rights  to  use)  the  training  materials,  perhaps 
to  generate  your  own  process  procedures? 


CMU/SEI-2003-TN-020 


13 


4  Summary 


Advanced  software  technologies  can  increase  productivity  and  reduce  costly  errors.  However, 
selecting  the  best  software  requires  understanding  a  number  of  factors.  The  TIDE  Program 
investigated  these  factors  during  the  course  of  selecting  MES  software  for  two  SMEs.  The 
effort  underscored  the  need  for  SMEs  to 

•  understand  their  business  and  how  the  proposed  software  will  support  their  firms’ 
growth  strategy 

•  use  a  process  to  understand  requirements  and  correlate  to  capabilities 

•  scale  the  selection  process  to  fit  the  organization 

•  involve  experienced  personnel  (including  outside  decision  support  and  technology 
adoption  specialists  if  necessary)  to  clarify  issues  and  identify  pitfalls 

•  investigate  vendors  and  their  products  from  a  variety  of  perspectives 
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Appendix  A:  Analytic  Hierarchy  Process 


The  Analytic  Hierarchy  Process  (AHP)  was  proposed  by  Saaty  over  20  years  ago  and  is  a 
widely  used  technique  for  multi-attribute  decision  making  [Saaty  80] .  It  is  a  method  of 
prioritizing  decisions  by  incorporating  relevant  decision  criteria.6  This  prioritization  achieved 
through  pair-wise  comparisons  of  competing  objectives  and  through  making  subjective 
judgments.  This  results  in  a  ratio  scale  of  relative  values.  The  AHP  is  carried  out  in  two 
phases.  In  the  Design  Phase,  a  criteria  hierarchy  is  set  up.  In  the  Evaluation  Phase,  pair-wise 
comparisons  are  used  to  evaluate  alternatives.  Figure  2  on  the  next  page  illustrates  the  major 
steps  involved  in  an  AHP  facilitated  evaluation. 


Structuring  the  Evaluation 

The  initial  step  in  using  the  AHP  tool  is  structuring  the  decision  to  be  made.  In  this  case,  the 
method  was  used  to  evaluate  and  eventually  recommend  a  COTS  product. 


Criteria  Development 

Criteria  are  statements  or  conditions  that  serve  to  validate  that  a  requirement  has  been  met. 
They  help  to  translate  the  subjective  to  a  more  objective  perspective.  Criteria  development 
can  be  a  layered  process  that  repeatedly  asks  “Why?”  or  “What  does  that  mean?”  This 
recursive  decomposition  must  be  used  with  caution  however;  it  is  very  easy  to  quickly  build  a 
model  that  becomes  cumbersome  in  future  steps. 


User  Requirements  Definition 

The  first  step  is  to  gather  key  stakeholders  to  brainstorm  user  requirements.  Figure  3  shows 
the  beginning  steps  of  establishing  user  requirements  for  a  COTS  software  product.  Three 
different  requirements  have  been  identified: 

1 .  All  functionality  is  provided. 

2.  Product  integrates  with  legacy  systems. 

3.  Product  is  “adoptable”  by  the  organization. 


6  This  is  a  generic  description  of  the  AHP  process  applied  to  software  resolution.  Appendix  B  reflects 
the  specific  requirements  matrix  that  was  used  in  this  case. 
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Figure  2:  AHP  Methodology  of  E valuation 
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Figure  3:  AHP  Tool  Capturing  Initial  User  Requirements 


User  requirements  definition  requires  more  than  brainstorming  a  wish  list  of  features  and 
functions.  Users’  needs  and  wants  must  be  identified  and  structured  to  facilitate  evaluating 
alternatives.  AHP-based  tools  provide  a  consistent  and  repeatable  process  for  translating 
requirements  into  evaluation  criteria. 
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Clustering 

One  of  the  more  time-  and  labor-intensive  aspects  of  the  evaluation  process  is  establishing  a 
list  of  criteria  in  such  a  manner  that  all  requirements  can  be  clearly  understood  and 
communicated  to  stakeholders  and  decision  makers.  This  task  is  aided  by  “clustering” — 
grouping  requirements  into  "theme  categories”  that  will  become  the  evaluation  criteria 
hierarchy.  Figure  4  shows  the  three  previously  mentioned  requirements  (functionality, 
integrability  and  adoptability)  as  the  theme  categories  containing  nine  different  evaluation 
criteria. 
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Figure  4:  Using  Clustering  to  Identify  Criteria 
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Identifying  Alternatives 

Once  high-level  user  requirements  and  related  evaluation  criteria  have  been  established, 
viable  alternative  COTS  products  can  be  identified.  At  this  stage,  the  evaluation  team  will 
frequently  incorporate  a  user  requirement  involving  the  strength  of  the  company.  Figure  5 
shows  how  an  AHP  tool  handles  the  list  of  alternatives  (COTS1,  COTS2,  COTS3,  and 
COTS4).  It  also  shows  the  user  requirement  “company  strength”  along  with  four  associated 
evaluation  criteria. 
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Figure  5:  Adding  Alternatives  and  Developer-Related  Requirements 
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Establishing  the  Evaluation  Hierarchy 

Once  the  alternatives  have  been  identified  and  a  sufficient  number  of  criteria  established,  the 
AHP  tool  can  automatically  create  the  evaluation  hierarchy.  At  this  stage  all  criteria  are 
equal;  no  attempt  has  been  made  to  establish  weights  or  priorities. 


Figure  6:  The  Initial  Hierarchy  for  Evaluation 
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Some  criteria  (e.g.,  “Has  short  learning  curve”)  may  require  additional  definition  or  another 
level  of  refinement.  The  following  screen  shows  the  addition  of  another  “branch”  to  the 
“adoptable”  portion  of  the  hierarchy. 
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Deriving  Requirement  and  Criteria  Weights 

Once  requirements  and  criteria  have  been  identified,  the  team  can  establish  priorities  for 
each.  The  process  is  established  through  the  mechanism  of  pair-wise  comparisons  in  which 
each  requirement  and  criterion  is  compared  against  its  “siblings”  within  the  evaluation 
hierarchy.  In  the  example,  the  high-level  requirements  that  will  be  compared  are 

•  degree  of  functionality 

•  ease  of  integration 

•  ease  of  adoption 

•  degree  of  company  strength 

Similarly,  within  the  theme  category  (requirement)  “adoptable,”  several  criteria  will  be 
compared: 

•  follows  Windows  conventions 

•  is  very  intuitive 

•  has  a  short  learning  curve 

In  this  example,  a  “verbal”  approach  has  been  used  to  compare  the  top-level  requirements. 
The  following  screen  capture  shows  the  comparison  matrix  between  the  four  user 
requirement  categories.  Initial  comparisons  have  been  made,  and  normalized  weights  have 
been  assigned  by  the  AHP  tool.  Note  that  the  tool  provides  a  histogram  to  show  the  relative 
importance  of  each  requirement  as  the  process  unfolds. 
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Figure  8:  Pair-Wise  Comparison  of  High-Level  Requirements 
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This  pair-wise  approach  allows  evaluators  to  compare  tangibles  or  intangibles  on  a  reliable 
scale.  Each  evaluator  expresses  an  opinion  and  all  individual  judgments  are  collected  and 
aggregated  into  a  group  judgment. 


Figure  9:  Evaluation  at  the  Completion  of  All  Comparisons 

At  this  stage  the  evaluation  team  has  established  weights  for  each  of  the  criteria  and 
expressed  its  opinion  of  the  four  COTS  software  packages  and  their  ability  to  satisfy  the 
different  criteria. 
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The  high-level  requirements  have  been  weighted  and  sorted.  Their  relative  importance  is 
illustrated  using  bar  graphs.  In  this  example,  the  evaluation  team  deemed  “functionality”  as 
by  far  the  most  important  requirement.  It  has  a  normalized  score  of  .480,  nearly  half  the  total 
assessment  of  utility.  In  other  words,  the  team  felt  that  functionality  was  nearly  as  important 
as  all  other  requirements  combined. 
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Figure  10:  Relative  Importance  of  High-Level  Requirements 


At  this  stage,  each  team  member  will  understand  how  other  team  members  feel  about  the 
requirements  and  criteria  and  how  their  different  perspectives  influence  the  evaluation. 
Further,  the  team  members  will  see  where  they  agree.  Effort  can  therefore  be  focused  on 
areas  of  disagreement  or  where  there  are  points  of  uncertainty  or  misunderstanding. 
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Sensitivity  Analysis 

Because  the  evaluation  process  is  inherently  uncertain,  it  must  accommodate  sensitivity 
analysis,  which  determines  what  influence  each  assumption  has  on  the  recommendation.  At 
each  level  of  the  hierarchy,  the  evaluation  team  can  see  the  relative  importance  of  its  criteria 
(left-hand  pane  below).  The  team  can  also  dynamically  change  these  relative  weights  and 
view  the  outcome  (right-hand  pane). 


11  Dynamic  Sensitivity  for  nodes  below:  Goal:  Evaluate  COTS  Software  Products  and  Select  One  Product 
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Sensitivity  w.r.t.:  Goal:  Evaluate  COTS  Software  Products  and  Select  One  Product 
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Figure  11:  Dynamic  Sensitivity  Analysis 
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“What  If?”  Scenarios 


An  important  use  of  dynamic  sensitivity  analysis  is  in  “what  if’  scenarios,  which  test  the 
robustness  of  the  recommendation  under  a  variety  of  different  assumptions.  Since  a  number 
of  the  criteria  in  a  typical  evaluation  may  be  quantifiable,  a  sensitivity  analysis  can  show  the 
extent  to  which  the  recommendation  might  change  if  an  assumption  were  altered. 

In  the  example,  products  COTS1  and  COTS4  have  virtually  the  same  ratings  (.265  vs.  .263). 
Increasing  the  importance  of  “functionality”  does  not  cause  the  relative  scores  of  COTS1  and 
COTS4  to  change.  The  recommendation  is  very  insensitive  along  the  dimension  of 
functionality.  Similarly,  the  importance  score  of  “integration”  must  substantially  increase 
before  it  changes  the  rank  of  the  products.  However,  increasing  the  importance  of 
adoptability  changes  the  recommendation  from  COTS1  to  COTS4.  Increasing  the  importance 
of  “company  strength”  only  serves  to  increase  the  score  of  the  leader,  COTS1. 

By  conducting  this  type  of  sensitivity  analysis,  the  evaluation  team  can  focus  on  issues  that 
can  potentially  change  the  recommendation:  the  criteria  in  the  “adoptability”  requirement. 
Examining  the  “adoptability”  branch  in  detail  provides  insight  into  the  recommendation.  As 
the  following  screen  capture  illustrates,  the  team  felt  that  COTS4  was  significantly  more 
intuitive  than  the  other  products.  This  rating  was  sufficient  to  award  COTS4  the  highest  rank 
in  this  dimension.  By  highlighting  these  criteria,  the  team  can  double -check  the  reasons  for 
the  initial  ratings  to  make  sure  that  all  team  members  are  comfortable  with  the  conclusions. 
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Figure  12:  Closer  Examination  of  the  “Adoptability”  Criteria 
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Building  Consensus 

As  with  criteria  comparisons,  the  evaluation  team  sees  where  it  agrees  and  disagrees  on  a 
product  evaluation.  Where  there  is  no  disagreement  on  where  the  decision  is  insensitive  to 
changes  in  assumptions,  there  is  no  benefit  in  protracted  discussion.  Where  there  is 
disagreement  or  where  the  decision  can  change  with  a  modest  change  in  assumptions,  it  is 
worth  the  team’s  time  to  scrutinize.  Building  a  consensus  within  the  evaluation  team  and 
communicating  such  to  the  stakeholders  is  arguably  the  most  valuable  aspect  of  this 
methodology. 
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Appendix  B:  AHP  Requirements  Matrix 


The  following  pages  contain  the  specific  AHP  requirements  matrix  that  was  generated  on  this 
project.  This  matrix  was  formatted  for  use  as  a  checklist  and  note  template  for  use  during 
product  demonstrations,  to  aid  in  Product  Management  (PM)  tool  selection. 


Table  1 :  User  Requirements  Review  of  Shop  Control  Software 


score 

Goal:  Select  a  PM  Tool 

Comments 

1 .  Production  Management  Functionality 

Provide  the  necessary  production 
management  functions  that  meet  the 
needs  of  a  typical  small  job  shop. 

1 .1  Engineering  Definition  Process 

Designer 

1 .1 .1  Engineering  Change  Notification 

1 .1 .2  Accommodates  legacy  definitions 

1.1.3  Travelers 

1.1.4  Routing 

1 .1 .5  Bill  of  Material  Mgmt. 

1 .2  Inventory  Mgmt. 

1 .2.1  Can  write  off  obsolete  items 

1 .2.2  Track  finished  goods  inventory 

1 .2.3  T rack  work  in  process  inventory 

1 .2.4  Inventory  Link  to  Orders 

1 .2.5  Different  Units 
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score 

Goal:  Select  a  PM  Tool 

Comments 

1 .3  Data  Collection  and  Dissemination 

1 .3.1  CAD  Interface 

Tool  can  view  AUTOCAD,  Unigraphics, 
or  other  popular  CAD  tools. 

1 .3.2  Customer-Supplied  Bar  Code 

Tool  can  import  or  interface  with 
customer-supplied  bar  codes. 

1 .3.3  Access  linked  drawings  and  text 
documents. 

i 

1 .3.4  Information  is  collected  and  validated 
automatically  to  improve  accuracy 

1 .3.5  Collect  Labor  Time. 

1 .3.6  Bar  codes  built  into  forms  and  reports 

1.4  Mgr.’ s  “Desktop” 

1 .4.1  Reports 

1 .4.1 .1  Standard  Tools 

Define  a  report  using  applications  such  as 
Crystal  Report,  Access,  Word,  or  Excel. 

1 .4.1 .2  Report  Templates 

Tool  comes  with  a  portfolio  of  standard 
reports  that  meet  most  of  the  company’s 
needs. 

1 .4.2  Executive  Information  System 

Provides  managers  with  fast,  easy 
executive-level  insight  into  important 
production  information. 

1. 4.2.1  Supplier  management 

1 .4.2.2  Planning 

Tool  enables  user  to  do  rough-cut  capacity 
planning,  material  planning,  “what  if” 
planning,  budgeting,  and  so  forth. 

1.4.3  Traceability 
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score 

Goal:  Select  a  PM  Tool 

Comments 

1.4.4  Quality  Management 

Tool  has  standard  statistical  process 
control  (SPC)  functions. 

1 .4.5  Keyword  Search 

Can  conduct  a  keyword  search  across  all 
data  files 

1 .4.6  Identification  Opportunities 

Facilitates  identification  (ID)  of 
opportunities  to  make  changes  to 
production  (paths,  schedules,  etc.)  that 
will  facilitate  better  factory  throughput 

1 .4.7  Alarms 

System  alarms  warn  of  certain  critical 
situations. 

1 .4.8  Proactive  Information  Management 

Proactive  management  of  information 
flow  to  customers,  vendors,  and  others 

1 .5  Integrated  Business  Functionality 

1 .5.1  Integrated  Accounting 

1 .5.2  Integrated  Purchasing 

1 .5.2.1  Blanket-  Order  Mgnit. 

1 .5.2.2  Alternate  Vendor 

1 .5.2.3  Receive  to  stock  or  to  job 

1 .6  Collaboration  Portal 

Plas  the  ability  to  publish  information  for 
use  by  customers  and/or  vendors, 
providing  insight  into  production 

1 .6.1  eBusiness  Interface 

Tool  supports  eBusiness  interfaces  with 
customers. 

1 .6.2  External  Viewer 

Provides  a  Web-enabled  viewer  (browser 
based)  to  enable  either  customer  or 
vendors  to  check  on  orders. 
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score 

Goal:  Select  a  PM  Tool 

Comments 

1 .6.3  Event  Management 

1 .7  Order  Management 

1 .7.1  Control  Material  Effectively 

1 .7.2  Search 

Can  search  for  orders  based  on  a  variety 
of  criteria 

1 .7.3  Order  Acknowledgement 

1 .7.4  Process  Orders  Efficiently 

1 .8  Schedule  Realistically 

1 .8.1  Basic  Infinite  Scheduling 

1 .8.2  Advanced  Scheduling 

Finite  and  “what  if”  scheduling 
capabilities 

1 . 9  Quote  accurately  and  easily 

The  tool  supports  easy  development  of 
estimates  and  quotes 

1.9.1  Quote  Tracking 

Provides  a  follow-up  reminder  and  the 
ability  to  save  and  archive  old  quotes. 

1 .9.2  Routers  and  Material  Sheets 

Development  of  quotation  results  in  the 
development  of  routers/travelers  for  the 
proposed  job. 

1 .9.3  Inventory  Link 

Quote  systems  in  linked-to  inventory. 

1 .9.4  Same-as-Except 

System  supports  use  of  previous  quotes 
or  jobs  to  develop  new  estimates  or 
quotations. 

1.9.5  Estimating 

Can  easily  develop  estimates  for  new 
proposals 
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score 

Goal:  Select  a  PM  Tool 

Comments 

2  Integrate 

Integrates  well  with  other  elements 
(software  and  hardware)  of  the  SME’s 
system 

2 . 1  Open  Database  Connectivity  (ODBC) 

2 . 2  Other  Legacy  Systems 

Can  easily  integrate  with  existing  systems. 

3  Sustainable 

New  software  tools  and  process  can  be 
adequately  sustained  by  the  SMEs. 

3 . 1  Company  Strength 

The  company  that  developed  the 
software  is  strong,  and  will  be  able  to 
support  the  product  and  produce  the 
appropriate  upgrades  and 
enhancements,  and  provide  long-term 
support  for  this  product. 

3.1.1.  Profitability 

3.1 .2  Market  Share 

3.1.3  Installed  Base 

3.2  Extensible 

The  developer  has  plans  for  future 
additional  features  or  attributes  to  keep 
up  with  evolving  needs  of  the 
manufacturer. 

3.3  Scalability 

The  product  can  “grow”  to  accommodate 
additional  users  and  a  greater  number  of 
files. 

3.4  Supported  Evolution 

The  product  will  be  supported  by  the 
developer  with  planned  enhancements  and 
upgrades  to  keep  it  technically  current. 

3.5  Support 
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score 

Goal:  Select  a  PM  Tool 

Comments 

3.5.1 

Documentation 

Tool  has  documentation  that  adequately 
enables  the  users  to  sustain  the  system. 

3.5.2 

Help  Desk 

3.5.3 

User  Groups 

3.5.4 

Local  Support 

Tool  has  local  technical  support  for  on¬ 
site  assistance. 

4 

Reliable 

New  software  tool  is  reliable. 

4.1 

Easy  Fixes 

Failures  can  be  fixed  by  the  SME’s 
personnel. 

4.2 

Failure  Consequences 

If  the  system  fails,  it  does  so  in  a  non- 
catastrophic  way  (data  is  not  lost,  the  failure 
does  not  bring  down  other  elements  of  the 
environment,  etc.). 

4.3 

MTTR 

The  mean  time  to  repair  any  failure  is 
adequately  short. 

4.4 

MTBF 

The  software  has  an  adequate  mean 
time  between  failures. 

5 

Adoptable 

5.1 

Good  Graphic  User  Interface  (GUI) 

5.2 

Implementation 

Can  be  installed  and  ready  for  use  in  a 
weekend 

5.3 

Tailorable  and  Flexible 

The  tool  is  easy  to  tailor,  using  standard 
templates,  to  meet  the  unique  needs  of 
most  users. 
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score 

Goal:  Select  a  PM  Tool 

Comments 

5.4 

Software  tool  is  intuitive. 

5.5 

Training 

6 

Operating  Environment 

6.1 

SQL  or  other  “easy”  database 

6.2 

Object  Linking  Embedding  Database 
(OLE  DB),  Java,  or  Distributed 
Component  Object  Model  (DCOM) 

6.3 

ODBC  Compliant 

6.4 

Windows  NT,  XP,  or  2000 

6.5 

PC  Based 
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Appendix  C:  Product  Dossier  Guidelines 

By  Edwin  Morris7 


Overview 

The  product  dossier  artifact  captures  all  the  information  regarding  a  single  COTS  product, 
including  characteristics  of  the  vendor;  product  architecture  and  functional  capabilities; 
standards  supported;  required  hardware  and  software  configurations;  nonfunctional 
characteristics  such  as  usability,  supportability,  reliability,  interoperability,  portability,  and 
scalability;  quality  of  documentation;  costs;  and  licenses.  A  Product  Dossier  is  created  when 
the  product  is  introduced  and  updated  as  appropriate. 


Purpose  of  Captured  Information 

The  product  dossier  artifact  accumulates  and  organizes  information  sufficient  to  record 

•  the  history  of  contacts  with  the  vendor  regarding  the  product 

•  the  history  of  consideration  and  use  of  the  product 

•  raw  (unfiltered)  information  about  a  product  and  product  vendor  gathered  directly  from 
the  vendor  (documentation,  claims,  price  lists,  demonstration  versions,  etc.),  and  from 
third  parties  (such  correspondence  and  reviews  by  other  users,  trade  journal  articles, 
business/financial  analysis,  etc.) 

•  processed  (filtered)  data  obtained  during  consideration  of  a  product  including  the  results 
of  investigations  into  the  product  and  vendor,  information  describing  the  exact 
configuration  of  the  product  evaluated,  and  data  gathered  during  evaluation  activities  and 
benchmarking 

•  the  analysis  of  the  product  and  vendor,  including  product/vendor  strengths,  weaknesses, 
related  products  and  ensembles,  and  architecture  or  usage  constraints  identified  during 
evaluation 

•  the  history,  rationale,  and  specific  activities  for  customization  and  tailoring  of  the  product 

•  the  history,  rationale,  and  specific  activities  for  integration  of  the  product 

•  the  history  of  version  releases 

•  the  history  and  rationale  for  upgrade  decisions  and  certification  activities 


7  These  guidelines  by  Edwin  Morris  of  the  SEI  first  appeared  in  Evolutionary  Process  for  Integrating 
COTS-based  systems  (EPIC)  [Albert  02].  Introductory  text  that  is  only  relevant  to  the  EPIC  process 
has  been  omitted. 
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Information  Needed 


The  goal  for  populating  the  Product  Dossier  is  to  capture  information  sufficient  to  select  (or 
rule  out)  a  specific  product  version,  to  maintain  data  about  the  architectural,  design, 
implementation  and  testing  ramifications  of  using  the  product,  to  transition  necessary  skills  to 
stakeholders  (such  as  maintainers  and  end  users),  and  to  support  the  maintenance/evolution 
process  of  the  product  in  the  system. 

The  categories  of  information  maintained  within  the  Product  Dossier  are  extensive.  Some  of 
this  information  is  developed  to  support  the  selection  of  the  product.  Other  information  is 
developed  as  the  product  is  incorporated  and  maintained  in  the  system.  Thus,  a  Product 
Dossier  is  a  living  document  that  represents  the  state  of  knowledge  about  a  product  during 
the  time  it  is  considered,  used  in,  and  maintained  for  the  system.  Examples  of  the  categories 
of  information  that  are  maintained  in  a  Product  Dossier  are  identified  below.  The  type  and 
degree  of  information  maintained  for  each  category  will  depend  on  a  number  of  factors, 
including  the  characteristics  of  the  product,  the  stage  in  product  selection  and  use,  and  how 
the  product  is  or  will  be  used  in  the  system.  In  addition  to  example  categories,  sample 
questions  that  illuminate  the  intent  of  the  categories  are  provided. 


Vendor  Characteristics 

Organizational  •  Has  the  organization  existed  in  its  present  foim  for  a  suitable  period 

stability  to  indicate  that  it  is  stable? 


Financial  stability 


•  Is  the  organization  making  money? 

•  What  are  the  financial  trends? 


N ationality  •  Is  the  organization  based  in  the  U.S.  or  a  nation  allied  with  the  U.S.? 

Ease  of  access  •  Is  there  sufficient  access  to  the  organization  for  answering  technical 

and  business  questions? 


Independence 


Reputation 


•  Does  the  vendor  make  independent  decisions,  or  is  it  (effectively) 
controlled  by  another  organization? 

•  Are  the  goals  and  directions  of  the  controlling  organization 
appropriate  for  the  needs  of  the  target  system? 

•  Does  the  organization  have  a  reputation  for  quality? 

•  Is  delivery  timely? 

•  Is  the  organization  responsive  to  customers? 
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Support  infrastructure  •  Docs  the  organization  otter  local  offices,  hotlines,  installation,  and 

integration  support? 

Engineering  approach  •  Is  the  engineering  approach  used  by  the  organization  appropriate 

and  compatible  with  the  customer’s  expectations  and  needs? 

Maintenance  approach  •  Is  the  maintenance  approach  appropriate  and  compatible? 

History  •  What  is  the  history  of  the  organization?  Where  did  the  organization 

come  from  and  how  did  it  come  to  market  this  product? 

Basic  Product  Characteristics 

Shipment  dates  •  When  was  the  product  first  made  available  to  customers? 

Product  stability  •  What  is  the  release  history  of  the  product? 

•  What  types  of  changes  were  made  for  various  releases? 

Install  base  •  How  many  copies  of  the  product  are  in  use? 

•  How  many  organizations  use  the  product? 

•  Are  these  organizations  similar  to  the  target  organization? 

•  Can  the  use  of  the  product  by  these  organizations  be  verified  (i.e„  not 
marketing  hype  or  shelfware)? 

Customer  references  •  What  customer  references  are  available? 

•  How  do  these  customers  use  the  product  when  did  they  take 
delivery,  how  many  copies  of  the  product  do  they  use,  and  how 
many  users  are  supported? 

•  What  are  their-  impressions  of  the  vendor,  product  support  and  so 
forth? 

•  Is  the  use  of  the  product  by  these  customers  similar  to  the  anticipated 
use  of  the  target  organization? 
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End-of-life  plans 


Availability  of 
training 


Access  to  hotline 


Consultants 


Delivery  method 


Standards 

DoD  standards 

Industry  standards 


•  What  phase-out  or  end-of-iife  planning  is  being  considered  by  the 
vendor  for  the  product? 

•  When  is  a  phase-out  or  end  of  life  planned? 

•  What  will  the  upgrade  path  be? 

•  What  will  this  upgrade  require  of  users? 

•  Are  any  plans  documented  and  available  to  customers? 

•  What  training  is  available  for  the  product,  when  and  where  is  it 
offered,  and  is  it  tailored  to  the  customers’  needs? 

•  For  what  groups  of  stakeholders  (system  personnel,  maintainers,  end 
users,  etc.)  is  training  available? 

•  Arc  any  third  patties  providing  training? 

•  During  what  hours  of  operation  is  a  hotline  available? 

•  What  types  of  support  are  available? 

•  Are  hotline  calls  fielded  domestically? 

•  Are  there  appropriate  capabilities  to  maintain  required  security? 

•  Are  vendor-sanctioned  consultants  available? 

•  Are  third-prut}'  consultants  available? 

•  What  is  the  availability  and  cost  for  consulting? 

•  What  media  is  used  for  delivery  of  the  product  and  product  upgrades 
(tape,  CD,  internet,  etc.)? 


•  What  Department  of  Defense  (DoD)-specific  standards  are 
supported? 

•  What  general  industry  standards  are  supported? 

•  What  standards  body  is  responsible  for  the  standard? 
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How  do  organizations  join  or  influence  the  direction  of  the  standard? 


Organizational 

Completeness 


Confidence 

Hardware 

Configuration 


Communications 


•  Ls  the  standard  widely  supported? 

•  Do  one  or  moie  organizations  have  extensive  control  over  the 
standard? 

•  What  is  the  release  history  of  the  standard? 

•  How  can  contact  be  made  with  the  group  or  committee  responsible 
for  the  standard? 

•  Does  the  product  and  vendor  meet  special  standards,  procedures,  and 
protocols  required  by  the  taiget  organization? 

•  Does  the  product  implement  a  subset  of  the  standard,  the  complete 
standard,  or  a  superset  of  the  standard? 

•  What  are  the  plans  for  updates  or  enhancements  to  subsequent 
versions  of  the  standard? 

•  How  is  standards  compliance  verified? 


•  What  are  the  minimal,  recommended,  and  maximum  hardware 
configurations  (computers,  processors,  memory,  disk,  bus, 
peripherals,  etc.)? 

•  What  incremental  steps  can  be  made  in  hardware  to  increase  the 
performance  and  storage  capacity  of  the  system? 

•  Does  the  required  hardware  configuration  conflict  with  that  of  any 
other  system  with  which  the  product  must  interact  or  be  collocated? 

•  Is  a  special  or  different  development  testing,  or  support  environment 
required? 

•  What  communications  infrastructure  is  required? 

•  What  bandwidth? 

•  What  configuration? 
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Hardware 

•  Are  there  any  known  compatibility  problems  between  the  product 

compatibility 

and  hardware  components? 

Accuracy 

•  Is  the  accuracy  of  all  hardware  components  within  the  required 

configuration  appropriate  for  my  needs? 

Security 

•  Is  the  security  of  all  hardware  components  within  the  required 

configuration  appropriate  for  my  needs? 

Reliability 

•  Is  the  reliability  of  all  hardware  components  within  the  required 

configuration  appropriate  for  my  needs? 

Vendor  characteristics 

•  Are  vendor  characteristics  for  all  hardware  components  within  the 
required  configuration  appropriate  for  my  needs? 

Product 

•  Are  the  characteristics  for  all  hardware  components  within  the 

characteristics 

required  configuration  appropriate  for  my  needs? 

Upgrade 

•  How  is  the  upgrade  of  a  hardware  component  tied  to  the  upgrade  of 
the  product? 

•  How  long  after  an  upgrade  of  hardware  is  a  product  upgrade 
generally  available? 

•  How  long  are  old  versions  of  hardware  supported  by  the  product? 

Software 

Operating  system 

•  What  operating  system(s)  are  required  (including  versions)? 

•  Are  the  performance  and  size  characteristics  appropriate  for  the 
needs  of  the  target  system? 

•  What  mechanisms  exist  to  identify  and  resolve  problems  related  to 
the  interface  between  the  operating  system  and  the  product? 

•  Who  is  responsible  for  identifying  and  resolving  the  problem? 

Communications 

•  What  communications  support  is  required  (including  versions)? 

•  Are  alternate  communications  capabilities  supported? 

•  Are  the  performance  and  size  characteristics  appropriate  for  the 

42 


CMU/SEI-2003-TN-020 


needs  of  the  target  system? 


Database 


Related  applications 


Compatibility 

problems 

Accuracy 

Security 

Reliability 

Vendor  characteristics 


What  mechanisms  exist  to  identify  and  resolve  problems  related  to 
the  interface  between  communications  capability  and  the  product? 

Who  is  responsible  for  identifying  and  resolving  those  problems? 

What  database  support  is  required  (including  versions)?  Are 
alternate  databases  supported? 

Are  the  performance  and  size  characteristics  of  the  supported 
database(s )  appropriate  for  the  needs  of  the  target  system? 

What  mechanisms  exist  to  identify  and  resolve  problems  related  to 
the  interface  between  the  database  and  the  product? 

Who  is  responsible  for  identifying  and  resolving  those  problems? 

What  other-  applications  are  required  (including  versions)? 

Are  there  alternates  for  these  applications? 

Are  the  performance  and  size  characteristics  appropriate  for  the 
needs  of  the  taiget  system? 

What  mechanisms  exist  to  identify  and  resolve  problems  related  to 
the  interface  between  the  related  applications  and  the  product? 

Who  is  responsible  for  identifying  and  resolving  those  problems? 

Are  there  any  known  compatibility  problems  between  the  product 
and  any  software  components? 

Is  the  accuracy  of  all  software  components  within  the  required 
configuration  appropriate  for  the  needs  of  the  taiget  system? 

Is  the  security  of  all  software  components  within  the  required 
configuration  appropriate  for  the  needs  of  the  taiget  system? 

Is  the  reliability  of  all  software  components  within  the  required 
configuration  appropriate  for  the  needs  of  the  taiget  system? 

Are  vendor  characteristics  for  all  software  components  within  the 
required  configuration  appropriate  for  the  needs  of  the  taiget  system? 
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Product 

•  Are  the  product  characteristics  for  all  software  components  within 

characteristics 

the  required  configuration  appropriate  for  the  needs  of  the  target 
system? 

Upgrade 

•  How  is  the  upgrade  of  a  software  component  tied  to  the  upgrade  of 
the  product? 

•  How  long  after  an  upgrade  of  software  is  a  product  upgrade 
generally  available? 

•  How  long  are  old  versions  of  software  supported  by  the  vendor? 

Usability 

Intended  use  and  users 

•  Who  are  the  intended  users  of  the  product? 

•  For  what  use  was  it  intended? 

General  operability 

•  How  hard  is  the  product  to  use? 

Skill  level  required 

•  What  skills  are  required  by  users? 

Responsiveness 

•  What  is  the  response  time  under  a  light  load?  Average  load?  Peak 
load? 

•  Can  response  times  be  tuned  or  improved? 

Robustness 

•  What  is  the  mean  time  between  failures  for  the  product? 

•  How  does  the  product  respond  to  erroneous  input  and  operator 
error? 

Help  capabilities 

•  What  help  capabilities  are  available  in  the  product? 

Error  assist/recovery 

•  How  does  the  product  assist  users  when  they  make  a  data  input 
error? 

•  How  does  the  product  support  users  in  recovery  from  erroneous 
input? 

Understandability 

•  Is  the  product  easy  to  understand? 

•  Are  common  usage  paradigms  employed? 
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Learnability 


Supportability 

Dependencies 


Upward  compatibility 


Site  installation 
support 


Site  operation  support 


Analyzability 


How  long  will  it  take  before  employees  will  be  proficient  with  the 
product? 


Does  the  product  make  use  of  any  component  or  capability  provided 
by  an  organization  other  than  the  vendor? 

To  what  extent  does  success  of  the  product  within  the  target  system 
depend  on  these  organizations? 

How  is  failure  of  a  component  produced  by  another  party  handled? 

How  would  subcontractors  fair  if  subjected  to  the  same  evaluation 
scrutiny  as  the  vendor? 

Have  all  versions  of  the  product  been  upward  compatible? 

Which  versions  have  not  been  and  why? 

What  steps  must  be  taken  when  a  new  release  of  a  product  must  be 
installed? 

Who  is  responsible  for  installation  of  the  product  on-site? 

Will  the  vendor  install  the  product? 

Is  there  extra  cost  for  this  service? 

Can  taiget  organization  personnel  install  the  product? 

What  skills  are  required? 

Will  the  vendor  provide  personnel  to  support  initial  operations, 
perform  standard  maintenance,  or  diagnose  errors? 

Does  the  product  indicate  to  users/operators  when  maintenance  is 
necessary  or  an  error  has  occuned? 

Does  the  product  provide  capabilities  to  analyze  performance? 
Locate  problems  or  bugs? 

If  capabilities  are  not  provided,  how  is  this  accomplished? 
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Replaceability 


Preventive 

maintenance 


Special  support 


Interoperability 

Data  model/format 


Support  for  data 
access 


Support  for  control 


If  the  product  must  be  replaced  with  another  commercial  product 
what  changes  would  be  necessary  to  the  system? 

What  activities  would  be  necessary  for  data  migration? 

Is  periodic  preventative  maintenance  inquired? 

How  frequently? 

What  activities  are  involved? 

Is  a  special  or  different  development  testing,  or  support  environment 
required? 

What  are  the  characteristics  and  components  of  that  environment? 
What  tools  are  required  or  suggested? 


What  data  model  and  formats  are  employed  by  the  product? 

Are  they  published? 

What  standard  are  they  based  on? 

What  other  products  support  the  same  data  model/formats? 

What  interfaces  or  techniques  are  available  to  access  product  data? 
What  effort  is  required  to  access  product  data? 

Is  the  granularity  of  data  access  appropriate  for  the  target  system? 

Can  the  product  be  invoked  by  other  applications?  How? 

What  is  the  granularity  at  which  the  product  can  be  invoked? 

Can  other  products  control  low-level  functions  that  might  be 
necessary  in  the  integrated  system  (for  example,  commit  to  a 
change)? 

Can  the  product  invoke  other  applications?  How? 
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Infrastructure  utilized 


Reliability 

Test  regimen 


Type/frequency  of 
faults 

Recovery  from  faults 


Benchmarking 

Experience 


Performance 

Benchmarking 


What  constraints  are  placed  on  these  invocations? 

How  can  the  execution  of  the  product  and  other  components  be 
synchronized? 

What  timing  concerns  may  arise? 

What  infrastructure  is  used  to  support  the  communication  of 
messages,  data,  and  control  sequencing  within  the  product? 

Can  the  infrastructure  be  used  by  other  system  components  to 
internet  with  the  product? 


How  is  testing  performed  by  the  vendor? 

Are  the  results  of  testing  independently  verified? 

Are  test  scripts  and  results  available? 

What  is  the  mean  time  between  failures? 

What  is  the  frequency  of  different  sorts  of  faults? 

What  is  the  error-handling  strategy? 

Is  there  journaling  of  faults? 

Are  all  faults  trapped  before  the  system  panics? 

Are  reliability  benchmarks  available  for  the  product? 

Are  any  claims  made  about  reliability? 

Do  systems  requiring  similar  reliability  to  the  target  system  use  the 
product? 

Which  ones? 


•  Are  performance  benchmarks  available  for  this  product? 
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Time -related  behavior 


Resource  behavior 


Surge  capacity 

Adaptability/flexibility 


Documentation 

Design  information 


Maintenance 


Are  the  results  of  these  benchmarks  suitable? 

Do  the  benchmarks  reflect  a  usage  situation  or  pattern  consistent 
with  that  expected  of  the  product  in  the  taiget  system? 

Does  the  product  exhibit  appropriate  time-related  behavior 
(throughput,  lack  of  deadlock,  thread-safety,  latency,  etc.)? 

Is  there  any  potential  for  time-related  interactions  with  other  system 
components?  Where? 

Have  these  interactions  been  evaluated  and  determined  to  be  within 
acceptable  limits  or  risk  levels? 

Does  the  product  make  appropriate  use  of  resources  (processors, 
memory,  devices,  etc.)? 

Is  there  a  possibility  of  contention  for  resources  with  other  system 
components? 

Have  these  contentions  been  evaluated  and  determined  to  be  within 
acceptable  limits  or  risk  levels? 

Does  the  product  have  the  capability  to  handle  increasing  loads  as 
expected  (e.g.,  increased  number  of  transactions,  increased 
complexity  of  processing,  increased  number  of  tracks)? 

Can  the  product  be  tailored  to  efficiently  handle  an  appropriate 
range  of  performance  expectations  (transaction  rates,  numbers  of 
tracks,  etc.)? 

How  is  this  adaptation  accomplished? 


Is  the  available  design  information  sufficient  to  determine  whether 
the  design  is  appropriate? 

Is  it  sufficient  for  determining  an  integration  strategy  with  other 
target  system  components? 

Is  the  available  maintenance  information  sufficient  for  installation? 
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information 

Training  materials 


Customization 


Quality 

Policy  on  reproduction 

Licenses 

Usage/maintenance 

Transferability  of 
license 

RT  licensing 


Routine  use? 

Preventative  maintenance? 

Fault  isolation  and  recovery? 

Are  training  materials  and  courses  available? 

Are  they  appropriate? 

Are  they  affordable? 

Do  they  cover  an  appropriate  set  of  stakeholders  for  the  target 
system? 

Are  training  material/courses  tailored  for  specific  stakeholders? 

Can  documentation,  training  materials,  design  information, 
maintenance  information,  and  so  forth,  be  customized  for  unique 
taiget  system  needs? 

What  is  involved  in  customization? 

What  will  it  cost? 

Is  the  quality  of  all  documentation  and  other  information 
appropriate? 

Can  materials  be  reproduced  as  needed? 


Are  standard-usage  maintenance  licenses  appropriate  for  the  taiget 
system? 

Are  license  terms  negotiable? 

Are  site  licensing  and/or  quantity  discounting  available? 

Are  licenses  transferable  to  other  operating  units  or  other  agents 
working  on  behalf  of  the  taiget  oiganization? 

Are  separate  licenses  necessary/available  for  development  and 
deployed  platforms? 
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•  What  are  the  terms  of  these  licenses? 

Data  rights  •  What  data  rights  are  included  in  the  standard  license? 

•  Are  they  appropriate  for  the  target  system? 

•  Must  additional  data  rights  be  negotiated? 

Escrow  •  Can  source  code  be  escrowed? 

•  What  are  the  costs  and  stipulations  of  that  escrow? 

•  Is  an  escrow  a  reasonable  precaution  for  this  system? 

Discontinuation  •  What  lights  docs  the  target  organization  have  if  the  product  is 

discontinued? 

Expiration  •  What  events  occur  when  a  license  expires? 

•  Is  there  any  notification  of  impending  expiration? 

•  Are  licenses  “time  bombed”? 

Functional  Capabilities 

Appropriateness  •  Docs  the  product  offer  appropriate  functional  capability? 

•  Is  this  functionality  provided  in  an  appropriate  manner 
(appropriate  process,  interfaces,  quality,  etc.)? 

Process  consistency  •  Arc  the  processes  supported  by  the  product  appropriate  for  our 

organization? 

•  What  internal  (our)  processes  must  change? 

•  How  will  this  change  be  accomplished? 

Industry  practices  •  Does  the  product  conform  to  best  industry  practice? 

•  How  was  this  determined? 

Completeness  •  What  proportion  of  the  intended  system  capability  does  the 

product  provide?  How  was  this  determined? 

•  What  is  the  gap  between  the  functions  necessary  in  the  taiget 
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system  and  those  supported  by  the  product? 


•  What  level  of  effort  will  be  required  to  provide  missing 
capabilities  or  enhance  deficient  capabilities?  How  should  this 
be  accomplished? 

Tailoring/customization?  •  Is  the  product  suitable  “out  of  the  box”  or  does  it  require  custom 

construction  of  scripts,  code,  tables,  and  so  forth? 

•  What  effort  is  involved  in  performing  this  customization?  Who 
will  perform  this  customization? 

•  Must  this  effort  be  repeated  in  order  to  incorporate  new  product 
releases? 

Excess  •  Does  the  product  offer  additional  functional  capability  that  will 

not  be  used?  Should  not  be  used? 

•  What  impact  does  this  additional  capability  have  on  resource 
requirements,  performance,  and  so  forth? 


•  What  architectural  paradigms  are  evident  in  or  employed  by  the 
product? 

•  Are  they  appropriate  for  the  target  system? 

•  Does  the  product  suggest  architectural  paradigms  for  the  target 
system? 

•  Does  the  product  impose  architectural  restrictions  on  the 
system?  Are  they  appropriate? 

•  What  is  the  impact  on  other  system  components? 

Product  Version  Data 

Version  ID  •  What  are  the  version  number  and  release  date  of  the  product? 

•  What  additional  information  is  needed  to  uniquely  identify  the 
product  (e.g.,  revision  number,  patch  number)? 
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Architecture 

Product 


System 


Version  documentation  •  Identify  all  product  documentation,  including  user  manuals, 

reference  manuals,  release  notes,  installation  instructions,  known 
bug  lists,  and  so  forth. 

Version  capabilities  •  What  new  features,  capabilities,  and  fixes  arc  provided  by  this 

uniquely  identified  product? 


Product/System  Relationship 


System  configuration 


System  adaptation 


Integration 


Tailoring/modific  ation 


What  system  configurations  does  the  product  work  with  (or  is 
part  of)? 

What  environment  variable  settings  are  required? 

What  specific  settings  are  required  for  networking,  memoiy, 
processes,  peripheral  devices,  and  so  forth? 

What  adaptation  and  settings  are  required  of  other  components  of 
the  system  in  order  to  work  with  this  product? 

What  (new)  assumptions  or  expectations  does  the  unique 
product  version  make  regarding  interaction  with  other 
components  in  the  environment? 

What  changes  must  be  made  to  the  assumptions  made  by  the  rest 
of  the  system  regarding  the  behavior  of  this  version? 

What  integration  guidelines  must  be  followed  and  specific 
integration  activities  undertaken? 

What  tailoring  or  modification  of  the  product  is  required? 

What  settings  are  required  for  product  variables? 

What  scripts,  tables,  schemas,  4GL  code,  etc.,  are  required?  Why 
are  these  required? 

Were  workarounds  considered?  Why  were  they  rejected? 

Has  the  tailoring/modification  been  approved  by  an  authoritative 
control  board? 
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•  Was  the  product  vendor  consulted?  What  was  the  vendor’s 
response? 

•  Will  tailorfirg/modificafion  affect  the  contract  in  any  way  (e.g., 
changes  in  license  fees,  changes  in  maintenance  practices  or 
responsibilities)? 

•  What  assurance  is  there  that  the  modified  version  will  become 
part  of  the  standard  commercial  offering? 

•  Who  has/will  perform  the  tailoring/modificafion? 

•  Is  all  applicable  test  data  and  verification  of  test  passage  under 
configuration  control? 
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Appendix  D:  Cost  Comparison  Spreadsheet 


The  following  page  contains  the  Cost  Comparison  Spreadsheet. 
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Table  2:  Cost  Comparison  Spreadsheet 


CMU/SEI-2003-TN-020 


Appendix  E:  Manufacturing  Execution  Systems  Product 
Survey 


The  following  pages  contain  the  Manufacturing  Execution  Systems  Product  Survey,  a 
product  comparison  matrix  to  aid  in  the  software  selection  process. 

This  matrix  was  a  snapshot  circa  December  2001  by  Craig  Schlenoff  of  NIST;  any 
commercial  product  identified  in  this  document  is  for  the  purpose  of  describing  a  software 
environment.  This  identification  does  not  imply  any  recommendation  or  endorsement  by 
NIST,  SEI,  CMU,  or  TIDE. 
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